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An  Operational  Analysis  of  System  Calibration 

Donald  P.  Gaver 

Naval  Postgraduate  School 

Hasan  B.  Mutlu 

AT&T  Bell  Laboratories 


Mathematical  models  are  proposed  for  studying  the  impact  of  mis-calibration 
upon  operational  effectiveness.  Methodology  for  assessing  the  system  effective- 
ness and  an  approach  for  optimizing  the  effectiveness  of  a  calibration  program 
are  examined.  The  theory  application  is  discussed  and  the  results  of  some 
specific  and  convenient  models  are  presented. 


1.   Introduction 

The  effectiveness  of  many  systems  depends  upon  the  degree  of  the  calibration  of  their  subsys- 
tems.   For  example,  a  ship  with  navigational  equipment  that  is  out  of  calibration  may  not  be 
able  to  locate  its  destination,  or,  in  the  case  of  a  Navy  ship,  locate  an  adversary.  If  a  Navy 
ship's  weapon  system  is  also  out  of  calibration  the  difficulties  will  be  compounded.   An  analo- 
gous problem  arises  in  connection  with  engine  de-tuning,  when  fuel  consumption  likely 
increases  and  performance  decreases,  and  with  drift  of  communication  systems.    Similar  rea- 
soning  also  applies   to  hardware   and/or   software   configurations   of   computer 
systems,    e.g.    in   factory   robotic   systems  where  mechanical      re-calibrations 
occur   and  are   controlled  by   software   systems.      The   detrimental   effect        of 
mis-calibration   is  well   recognized;    the   systems   are   taken      to      ranges        or 
other   facilities    for   testing   and   recalibration. 
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The  purpose  of  this  paper  is  to  develop  simple  mathematical  models  to  investigate  ways  of 
dealing  with  mis-calibration.  If  the  various  aspects  of  the  problem  can  be  assembled,  some  gui- 
dance is  then  available  to  deal  with  it  effectively.  Although  many  realistic  elements  of  the 
problem  can  be  considered,  the  fundamental  issue  is  this;  given  that  important  subsystems 
depart  from  calibration  and  effectiveness  as  time  passes,  it  is  desirable  to  determine  a  schedule 
for  re-calibration  to  optimize  system  operational  effectiveness.  Frequent  calibration  of  impor- 
tant systems  would  be  highly  desirable  if  this  were  a  cost-free  operation.  However,  in  reality 
the  operational  cost  of  calibration  is  time  during  which  the  system  is  unavailable  for,  or  so 
degraded  as  to  be  incapable  of  adequately  performing,  its  operational  purpose.  It  should  be 
mentioned  that  the  actual  dollar  cost  in  man-hours,  of  performing  calibration  and  the  degrada- 
tion of  the  calibration  equipment  could  be  considered,  but  these  aspects  are  ignored  as  secon- 
dary effects. 

The  problem  discussed  here  is  recognizably  analogous  to  those  formulated  and  analyzed  in 
several  other  areas.  Similar  issues  arise  in  industrial  quality  control;  see  Lorenzen  and  Vance 
(1986),  who  utilize  renewal-reward  theory,  and  Smith  and  Vemuganti  (1968),  who  invoke  stan- 
dard Bayesian  decision  theory.  The  important  area  of  system  availability  monitoring,  in  both 
military  and  civilian  nuclear  power  industry,  requires  a  similar  analysis;  Thomas,  Jacobs  and 
Gaver  (1987)  recently  formulated  such  problems  and  solved  them  using  dynamic  programming 
techniques.  But  the  present  paper  is  intended  to  provide  a  skeletally  simple  model  for  a  partic- 
ular situation,  leaving  mathematical  elaboration  aside.  Furthermore,  we  emphasize  operationally 
relevant  costs  and  penalties  (i.e.,  times  out  of  service),  and  not  simple  monetary  costs,  although 
such  can  be  brought  in. 

Figure  1  is  an  idealized  graph  of  operational  effectiveness  against  time.  The  periods  of  duration 
C  denote  those  periods  during  which  the  system  has  zero  effectiveness  because  it  is  undergoing 
calibration  and  is  therefore  out  of  the  operational  area,  and  the  periods  of  duration  T  represent 
those  periods  during  which  the  system  is  operational,  but  diminishing  in  effectiveness.  The 
graph  suggests  that,  if  effectiveness  drops  with  time,  there  will  be  an  optimal  value  for  T,  a 
best  period,  T* ,  at  which  to  calibrate.  We  now  present  how  such  a  period  may  be  determined. 
More  complex  and  realistic  models  are  also  introduced. 


Figure  1.  Idealized  Graph  of  Operational  Effectiveness. 


2.  Mathematical  Models 

For  a  simple  mathematical  treatment  let  e(t)  be  the  effectiveness  (e.g.,  the  probability  of  suc- 
cessful operation),  at  time  t  after  the  calibrated  system  returns  to  service.  Let  C  be  the  time 
required  for  calibration,  and  T  the  duty  or  on-station  time.  Then  the  average  effectiveness  over 
a  cycle  of  length  T  +  C,  and  hence  in  the  long  run,  is 


e{T)  = 


\  e(t)dt  +  0 


T  +  C 


(2.1) 


the  term  0  represents  and  emphasizes  the  total  lack  of  effectiveness  during  the  calibration 
period.  In  order  to  maximize  e(T)  it  is  useful  to  study  the  derivative 
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r 

(T  +  C)e(T)-  je(t)dt 

WQ-  = j5 (2.2) 

rfT  (7+C)2 


as  it  depends  on  7:  if  de(T)/dT  =  0  for  T*  >  0  then  7*  is  a  candidate  for  a  time  between  the 
end  of  one  calibration  and  the  beginning  of  the  next.  Equivalently,  equation  (2.2)  asks  if  there 
is  a  positive  solution  T* ,  of 


e(T)=-±-  je(t)dt  (2.3) 


for  fixed  positive  C.  The  fact  that  such  a  solution  always  exists  and  that  it  defines  an  optimum 
can  be  established  from  the  usual  second  derivative  criterion.  Since  the  optimal  T  satisfies 
equation  (2.3),  it  turns  out  that  at  the  optimum  the  average  effectiveness  over  an  entire  cycle 
equals  the  effectiveness  at  the  time  the  active  part  of  the  cycle  ends;  or  symbolically  as  fol- 
lows: 


e(T)  =  e(T)  (2.4) 


where:  the  over-bar  signifies  the  time  average  of  effectiveness  over  T*  +  C.  All  of  the  above 
tacitly  assumes  that  the  function  e(t)  is  regular  enough  for  the  mathematical  operations 
invoked:  e(t)  >  0  decreasing  and  twice-differentiable  is  more  than  adequate;  if  e(t)  is  a  proba- 
bility then  e  (r)  <,  1 . 

To  increase  understanding,  we  will  examine  some  simple  specific  models. 
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2.1   Linear  Effectiveness  Loss 
Let 


\\-at,  0<,t^a~x 

««-    0.  a-*  Si  (25) 


so  that  the  downward-sloping  parts  of  the  graph  of  Figure  1  are  strictly  linear.  Then  equation 
(2.3),  the  equation  for  optimal  T  =  T* ,  is  as  follows: 


l-^=y^(r-|72),  0<T<a'1.  (2.6) 


It  is  clear  that  no  value  of  T  >  a  l  can  be  optimum.   The  equation  (2.6)  simplifies  to  the  qua- 
dratic as  follows: 


aT2  +  2aCT  -  2C  =  0  (2.7) 


with  a  single  positive  solution 


T*  =-C  +  ^C2  +  2Cla  (2.8) 


at  which  the  optimum  value  of  effectiveness  is  as  follows: 


\e(t)dt 
e(T-)=—. =  l-ar 


T    +C 
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=  (1  +  aC)  -  ^a2C2  +  2aC 

=  (1  +  aC)  -  ^(1  +  aCf  -  1  .  (2.9) 


It  is  interesting  that  the  solution  depends  only  upon  the  parameter  aC,  the  product  of  calibra- 
tion drift  rate,  a,  and  the  length  of  the  re-calibration  period,  C.  For  instance,  if  aC  — >  0  then 
effectiveness  approaches  unity  if  the  rate  of  calibration  degradation,  a,  approaches  zero,  or  the 
calibration  time,  C,  approaches  zero,  or  both,  or  one  approaches  zero  more  rapidly  than  the 
other  gets  large.  Alternatively,  this  shows  that  equal-effectiveness  or  a-C  tradeoff  curves  are 
simple  hyperbolas  in  the  (a,C)  plane.   A  tendency  in  this  direction  may  well  be  common. 

The  above  model  is  rather  crude,  but  is  easy  to  understand.  It  resembles  the  tool  wear  problem 
of  Smith  and  Vemuganti  (1968)  without  the  Bayesian  refinements,  and  with  operationally 
relevant  costs.  The  following  model  (Linear  Degradation  with  Diffuse  Damage)  is  more  quali- 
tatively appealing. 


2.2   Linear  Degradation  with  Diffuse  Damage 

Consider  a  more  specific  model  for  effectiveness,  one  that  relates  to  damage  inflicted  on  a  tar- 
get after  time  t  has  elapsed,  and  the  system  has  developed  an  (unsuspected)  bias  of  magnitude 
at.  At  this  time,  the  x-y  error  made  in  locating  a  target  is  assumed  to  be  given  by  the  joint 
Gauss/normal  density  as  follows: 


/(W)=^"eXP 


1  (x-atf     1  (y-aty 


2      a2         2      o2 


(2.10) 


If  a  cookie-cutter  damage  function,  with  radius  R,  is  in  effect  (0%  effective  if  x2  +  y2  >  R2, 
100%  effective  if  x2  +  y2  Z  R2)  then: 


e(t)=      \\    f(x,y;t)dxdy  . 

x2+y2SR2 
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However,  this  is  difficult  to  work  with,  and  even  overly  simplistic.  Instead,  suppose  that  a  von 
Neumann-Gauss  diffuse  damage  function  can  be  used;  (i.e.,  that  the  probability  of  critical  dam- 
age to  a  target  located  at  (0,0)  by  a  weapon  with  impact  point  (x,y)  is  equal  to  §(x,y)  =  exp(- 
a(x2  +  y2)))  (Eckler  and  Burr  (1972)).  Then  the  following: 


e(t)  =  jJ6(x,y)f(x,y,t)dxdy 


=  -^f  I  lexp[-a(x2+y2)]f(x,y;t)dxdy 


(     1      7        |     I  iat-xf 


exp  - 


2jco-«  2      o2 


exp[-ca2]<£t)2  (2.11) 


by  virtue  of  the  symmetry  assumed;  almost  free  of  charge  we  can  consider  asymmetrical  dam- 
age functions,  but  the  opportunity  is  declined.  The  above  integral  is  evaluated  at  sight:  it  is 
seen  to  be  essentially  the  convolution  of  two  normal  densities.  After  squaring,  as  demanded  by 
equation  (2.11),  the  result  is: 


l/2a 

e(t)  =  — = exp 

(a2+l/2a) 


{at)  (2.12) 


(o2+l/2a) 


Instead  of  dropping  off  linearly,  as  in  the  previous  case,  eft)  first  diminishes  rather  slowly,  later 
falling  quite  rapidly  (exponentially  fast)  towards  zero:  by  the  time  at  =  ^o2  +  l/2a,  effective- 
ness eft)  is  just  below  40%  of  its  maximum,  while  if  at  =  0.5^ a1  +  l/2a,  effectiveness  e(t)  is 
about  78%  of  the  maximum;  finally  if  at  =  0.25W  +  l/2a,  effectiveness  eft)  is  94%  of  the 
maximum.  Note  that  the  maximum  effectiveness  is  (1  +  2ao2)-1  <,  1;  if  either  a2  or  a  become 
large,  meaning  that  if  either  weapon  effectiveness  falls  off  rapidly,  with  miss  distance  (a  large), 
or  the  ultimate  weapon  delivery  variance  is  great  (a2  large),  then  even  maximum  effectiveness 
is  low. 
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In  order  to  solve  for  the  optimum  T*  write  (from  equation  (2.3))  the  following: 


(r+C)exp 


iflTY 


(cr2+l/2a) 


T 

=  Jexp 
o 


(at)2 


tf+ma). 


dt 


(2.13) 


Change  the  variables  to  the  dimensionless  version  as  follows: 


t  =  (a7)/(o2+l/2a)1/2  ;  y=  (aC)/(o2+l/2a),/2  , 


(2.14) 


so  one  can  solve  the  following  dimensionless  equation,  once  and  for  all,  for  t*: 


(x-rf)exp  (-T2)  =  J  exp  (-z  2  )dz  ; 


(2.15) 


the  positive  value  of  x,  namely  t*,  that  satisfies  this  equation  may  be  located  by  Newton- 
Raphson,  or  even  graphically:  one  can  plot,  for  given  y, 


L(i)  =  (x+7)^P(^2) 


and 


/?(!)=  \exp(-z2)dz 


on  the  same  piece  of  paper,  vs.  t. 


The  arbitrary  selected  y  values  and  the  corresponding  x*  values  from  the  computer  program, 
which  solves  the  dimensionless  equation  (2.15),  are  shown  in  Table  1. 


Y 

0.01 

0.02 

0.03 

0.04 

0.05 

0.06 

0.07 

0.08 

0.09 

X* 

0.236 

0.302 

0.346 

0.381 

0.410 

0.434 

0.456 

0.476 

0.494 

TABLE  1.  y  and  x*  Values. 


In  order  to  redefine  x  and  y  in  equation  (2.14)  and  the  effectiveness  formula  equation  (2.12)  in 
a  more  meaningful  form,  again  change  variables  to  the  following: 


v  =  l/2a;      p  =  — = — ;       k  =  a/o 
cr+v 


(2.16) 


where:  v  and  p  might  be  called  vulnerability  and  probability  of  success  respectively,  and  k  is 
constant.   So  one  can  write  the  following: 


x  =  kT^p  ;  y  =  kC^p 


(2.17) 


We  focus  attention  on  the  representation  equation  (2.12)  in  what  follows,  mainly  for  analytical 
and  computational  convenience: 


e(t)=pexp[-(ktf(l-p)]. 


(2.18) 


Thus,  the  preceding  expression,  at  the  optimum  point  T* ,  leads  to  the  following  relationship: 
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e(r)=pexp(-<x+)2)  (2.19) 


and  consequently  from  equation  (2.17),  the  optimal  proportion  of  on-station  time  can  be 
obtained  as  follows: 


y      c 


T*  =  — £—  ,  (2.20) 

ie*\-p 


so  the  proportion  of  on-station  time  is,  under  optimum  conditions,  as  follows: 


(2.21) 


T  +C       T*+Y 


Since  the  optimum  t  values  are  available  from  Table  1,  one  can  easily  cal- 
culate the  optimal  effectiveness,  (2 .4) (hereafter  called  simply  effective- 
ness) given  some  constant  variance  and  V  or  only  p.  Some  of  the  results 
are  tabulated  in  Table  2  as  an  example.  Note  that  the  effectiveness  de- 
creases as  the  variance  increases  while  V   is  held  constant. 

Example:  Suppose  a  =  1.5  yds/month,  C  =  0 .5  month,  o2  =  20  (yds)2  and  p  =  0.9  are  given. 
First  find  y  from  equation  (2.17)  as  0.053,  then  look  up  the  corresponding  x*  value  from  Table 
1,  which  is  0.417.  From  equation  (2.20)  T*  is  3.93  months,  from  equation  (2.21)  the  propor- 
tion of  on-station  time  is  88.7%,  and  from  equation  (2.19)  an  average  effectiveness  of  75.6% 
can  be  obtained. 
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Effectiveness 

(02=10;  v=200) 

0.901 

0.869 

0.845      0.824 

0.805 

0.789 

0.773 

0.759 

(02=20;  v=200) 

0.860 

0.830 

0.806      0.786 

0.768 

0.753 

0.738 

0.725 

TABLE  2.  Effectiveness  for  Constant  Variance  (o2)  and  v. 


2.3  Linear  Degradation  with  Diffuse  Damage  Using  Random  Drift 

As  an  alternative  model,  that  incorporates  the  possibly  different  drift  rates  of  different  indivi- 
dual systems,  suppose  that  the  drift,  a,  is  a  random  variable  with  an  appropriate  distribution 
function  instead  of  a  constant  as  in  equation  (2.12),  namely,  the  effectiveness  conditional  on  a 
is  as  follows: 


,     ,  l/2cc 

e{t\a)  =  — = exp 

(o2+l/2a) 


(at)2 


(o2+l/2a). 


(2.22) 


Then  the  expected  average,  or  unconditional  effectiveness  over  a  cycle  of  length  T  +  C,  in  the 
long  run,  becomes  the  following: 


E[e(T,a)]  = 


JE[e(.t\q)]dt 


T  +  C 


(2.23) 


In  order  to  be  specific,  and  also  so  that  explicit  mathematical  results  are  obtained,  let  a    have  a 
gamma  distribution  function  with  parameters  X  and  p\  as  follows. 
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/a2(x;A.,P)  = 


r(p) 


x.  p>o 


(2.24) 


and  use  the  following  for  fixed  a2,  (i.e.,  the  square  of  the  drift  rate  away  from  calibration), 


E<g2)-«2-£; 


Var(a2)  = 


"X2     p 


(2.25) 


Now  use  equation  (2.22)  and  (2.23)  to  obtain  the  following: 


E[e(t;q)]  = 


l/2a 


(o2+l/2a)  J 


exp  -x 


a2+l/2a 


/a2(x;X,P)dx 


After  substituting  the  gamma  density  function,  it  is  easy  to  see  that  the  result  of  the  integration 
yields  the  following: 


E[e(t;q))  = 


l/2a 


(o*+l/2a) 


U+ 


o2+l/2a 


(2.26) 


or  equivalently,  in  view  of  equation  (2.25)  the  following: 


E[e(t;q))  = 


l/2a 


(o2+l/2a) 


IP 


1+ 


(at? 


(cr2+l/2a)p 


(2.27) 


Various  analytical  properties,  of  the  previously  described  model,  will  now  be  recorded.  These 
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properties  provide  useful  insights  into  the  behavior  of  the  effectiveness  at  time  t. 
1.  If  P  ->  oo,  equation  (2.27)  becomes  the  following: 


E[e(t;a)]  =      1/2(*     expRar)2] 
cr+l/2a 


(2.28) 


which  reflects  the  fact  that  if  (3  increases,  the  variance  of  a  in  the  distribution  of  drift  rate 
decreases  towards  zero,  and  the  situation  reduces  to  that  of  previous  model. 

2.  If  (3  ->  1,  then  the  following: 


E[e(t;a))  = 


l/2cc 


(a2+l/2a+(a/)2)  ' 


(2.29) 


which  is  larger  than  the  effectiveness  in  the  equal-drift  case. 

In  order  to  solve  for  the  optimum  T*,  write  the  following  for  the  general  case: 


T+C 


1+ 


(aT)2 


>     (02+1/200(3 . 


=  J 


1+ 


(at)2 


{     (o2+l/2a)Pj 


-dr 


(2.30) 


Oiange  the  variables  as  follows: 


x  =  (ar)/(o2+l/2a)1/2  ;  y=  (aC)/(o2+l/2a),/2  , 


(2.31) 


so  that  one  can  solve  the  dimensionless  equation  as  follows: 
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x-fy 

'4 


-J 


dz 


(2.32) 


1+- 


the  positive  value  of  x,  namely  X*,  that  satisfies  this  equation  for  any  constant  P  may  be  found 
by  a  computer  program.  In  fact,  one  may  get  the  solution  for  the  special  case  p  =  1  by  making 
use  of  arctg  integration  for  the  right-hand  side  as  follows: 


Y  =  {arctg  x)  (1  +  x2)  -  x  . 


(2.33) 


In  general,  the  right-hand  integral  can  be  transformed  to  the  integral  of  a  Student's  t  density, 
and  the  t-tables  found  in  most  statistics  books  can  be  used  to  evaluate  it. 

The  arbitrary  selected  y  values  and  the  corresponding  x*  values  for  P  =  1  are  presented  in 
Table  3. 


Y 

0.01 

0.02 

0.03 

0.04 

0.05 

0.06 

0.07 

0.08 

0.09 

X* 

0.239 

0.307 

0.355 

0.392 

0.424 

0.451 

0.476 

0.499 

0.520 

TABLE  3.  Y  and  x*  Values  (p  =  1). 


At  this  point,  it  is  very  easy  to  calculate  the  effectiveness  given  some  constant  variance  and  v 
or  only  p  from  equation  (2.19).  Some  of  the  results  are  listed  in  Table  4  as  an  example.  As  in 
the  previous  case,  effectiveness  decreases  as  the  variance  increases  when  v  is  held  constant 
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Effectiveness 

(a2 =20;  v=150) 

0.835 

0.806 

0.784      0.765 

0.748 

0.733 

0.719 

0.706 

(02=30;  v=150) 

0.788 

0.761 

0.740      0.722 

0.706 

0.692 

0.679 

0.667 

TABLE  4.  Effectiveness  for  Constant  Variance  (a2)  and  v  (P  =  1). 


3.   Conclusions 


The  purpose  of  this  paper  has  been  to  show  that  quite  simple  mathematical  models  can  provide 
useful  ways  of  studying  the  impact  of  mis-calibration  upon  operational  effectiveness.  Metho- 
dology for  assessing  the  system  effectiveness  and  an  approach  for  optimizing  the  effectiveness 
of  a  calibration  program  have  been  examined.  We  have  concentrated  here  on  specific  and  con- 
venient models,  but  it  is  obvious  that  other  more  elaborate  mathematical  models  can  be  treated 
similarly.  The  relative  effectiveness  of  different  system  configurations  can  also  be  investigated. 

The  possibility  exists  that  operational  data  will  reveal  different  underlying  distributions,  and 
may  suggest  alternatives  for  evaluating  effectiveness  other  than  the  ones  described  in  this 
paper.  This  possibility  could  be  profitably  investigated. 
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